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Abstract 

This experiment tested the effect that calculator use had on 200 randomly assigned college students’ 
mathematical performance. The purposes of the current experiment were twofold: to measure the level of 
mathematical preparation of current college students, and to test whether calculators improve mathematical 
performance in such students as it does in precollege students. There was a significant difference in math 
performance between the two groups, with the calculator group outperforming the no-calculator group. The 
dozen demographic variables measured did not significantly predict performance. The improved mathematical 
improvement may not be due to calculators being easy but rather to the effect of learning experiences. 
Classroom policies concerning the use of calculators need to be evaluated and possibly changed. 
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Introduction 

Educators have voiced concerns regarding the level of mathematical preparation of students in general and those 
entering college in particular. The role of calculators in mathematical education has likewise been debated, 
despite the U.S. National Council of Teachers of Mathematics’s recommendation to use calculators at all school 
grade levels for instruction, homework, and assessment (NCTM, 1989). In 1992, D’Amore used 10 questions 
drawn from a third grade math test used by Hume (1932) to test ninth graders’ mathematical skills (Cornwall, 
1999). Only 25% of the sample passed the test (defined as obtaining 10 out of 10 correct; a perfect score). The 
following year he tested 2,436 Canadian students in grades five through seven and found similar results. In 
2006, Standing used D’Amore’s test (DT) as a test of mathematical ability, gave it to 75 university students at a 
selective liberal arts college in Canada, and found that only 33% of them passed it. Standing went on to 
investigate possible predictors for the students’ varied performance on the DT. He found that as their year in 
college increased, DT scores (number correct out of 10) decreased. He also found differences across students’ 
major, with those majoring in the natural sciences having the greatest number of perfect scores and those 
majoring in business having the lowest. Performance was positively related to self-rated mathematical ability, 
and to the predictions that each student made about his/her individual performance prior to taking the test. 
Standing found no relationship between DT performance and age, sex, GPA, liking of mathematics or science, 
background high school preparation, self-prediction of performance, self-rated mathematical ability, and liking 
of computers in general. 

These studies specifically prohibited the use of calculators to solve the problems, following the pedagogical 
assumption that students should be able to perform certain mathematical operations without the aid of 
calculators. Nonetheless, there has been some research on the effect of calculator use in solving mathematical 
problems. Hembree and Dessart (1986) analyzed the results of 79 research studies focused on precollege 
mathematics, and in all but for grade 4, the use of calculators improved the students’ basic skills in problem 
solving. These students also had a better attitude toward mathematics and more self-confidence in mathematics. 
Ellington (2003) meta-analyzed 54 research studies on calculator use by precollege students, and also found that 
students’ operational skills and problem-solving skills improved when calculators were allowed in the testing 
environment. She confirmed that the calculator group had better attitudes toward mathematics. 

Hanson, Brown, Levine, and Garcia (2001) compared the difference between standard calculators that were 
provided by the experimenter vs. those brought by 8th grade students, and found that performance was 
significantly higher for the group of students who used their own calculators as opposed to those who were 
given calculators with which they were not familiar. Bridgeman, Harvey, and Braswell (1995) did not find that 
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SAT test-takers benefited from using calculators, but Scheuneman, Camara, Cascallar, Wendler, and Lawrence 
(2002) found small but significant differences in performance (i.e., 73 points higher with calculators). Collins 
and Mittag (2005) did not find that the mathematical performance of undergraduate students in statistics classes 
improved with the use of calculators. Indeed, it seems students view calculators as a tool to circumvent the need 
to learn to think about and understand a mathematical problem (King & Robinson, 2012). 

College students who want to major in fields such as psychology, sociology, political science, business, and 
engineering, are often required to take college-level Statistics courses that require Arithmetic and Algebra I pre¬ 
requisites. There is some literature questioning the need for Algebra I as a pre-requisite (Sibulkin & Butler, 
2008), but it is unclear whether college students even have the arithmetic background necessary for college 
Statistics. Furthermore, if students today are taught mathematics with calculators from the early grades on, as 
specifically indicated by the Common Core State Standards Initiative (2015), their approach to, and method of, 
problem solving might very well revolve around the use of this tool. 

The purpose of the current experiment was twofold: to measure the level of mathematical preparation of current 
college students, and to test whether calculators (which Standing (2006) did not allow) improve mathematical 
performance in such students as it does in precollege students. If there is an improvement, it may not be due to 
calculators being easier, but rather due to students today learning with calculators as a part of their environment. 
In addition to determining the mathematical preparation of current college students, assessing their performance 
with and without a calculator will allow us to determine whether the errors they make are conceptual or 
computational/mechanical. We defined conceptual errors as those that involve a lack of understanding of 
arithmetic (e.g., not knowing which operations are needed to answer a specific question, not knowing how to 
sequence operations), while computational errors referred simply to arithmetic mistakes, like not knowing the 
multiplication tables. The researchers predicted that the students’ mathematical preparation would be low, and 
that as with precollege students, college students using calculators would also perform better than those not 
using them. 


Method 

Participants 

Participants consisted of 200 undergraduate students enrolled in multiple undergraduate psychology courses 
offered at a private, Mid-Atlantic liberal arts college. Some faculty members from those courses offered the 
students extra credit for participating. Eighty-one per cent (N = 161) of the participants were psychology 
majors. Eighty-three per cent (N = 165) were female and 17% (N = 34) were male, ranging in age from 18 to 
57, with a mean and median age of 21 and 20, respectively (87% were 23 or younger). Twelve per cent (N = 24) 
were 1 st year students, 25% (N = 49) were 2 nd year students, 42% (N = 83) were 3 ld year students, and 21% (N = 
41) were 4 th year students. 

Materials and Procedure 

Subsequent to obtaining approval from the college’s Institutional Review Board, the researchers gave a 
convenience sample of students an informed consent form prior to their participation in the study. After reading, 
signing, and returning it, the researchers randomly assigned the students to one of two groups: an experimental 
group that was allowed to use a basic calculator given to them, and a control group that could only use pencil 
and paper. Participants from both groups were allowed to use scratch paper for their work, and there were no 
time limits to the test. All participants received two sheets. The first sheet was the D’Amore Test (DT), 
consisting of 10 elementary arithmetic, constructed-response questions covering addition, subtraction, 
multiplication, division, fractions, and decimals (see Appendix), typical of topics reviewed in the Arithmetic 
pre-requisite course for college Statistics. The second sheet included 13 demographic questions for the students 
to fill out; seven of them were background questions, and the remaining six were 10-point Likert scale questions 
asking them to rate how much they liked math, how good their high school math instruction had been, how 
comfortable they were with computers, and to rate their natural mathematical ability, math anxiety, and interest 
in science, medicine, and technology. Once students had taken the DT, filled out the demographic questions, 
and turned in their two sheets, the researchers gave them a debriefing form that explained the full purpose of the 
study. 
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Results 

The participants’ average GPA was 3.14 and they reported having taken an average of six high school and 
college math courses. Fifty-four per cent (N = 109) of the participants reported believing they answered all 10 
questions correctly. Forty three per cent (N = 85) reported liking mathematics, 57% (N = 114) reported that 
their high school mathematics instruction was good, 86% (N = 172) reported feeling comfortable with 
computers, 57% (N = 114) reported having a natural math ability, 52% (N = 104) reported feeling math anxiety, 
and 46% (N = 92) reported being interested in science, medicine, and technology. 

The researchers coded each individual question of the DT as correct or incorrect, and then computed a total 
score (out of 10) for each student. The students’ scores ranged from 3 to 10, and only 45% of the entire sample 
answered all 10 questions correctly, revealing insufficient preparation for college level Statistics. In the 
calculator group 56% (58/103) had a perfect score, and in the no-calculator group 33% (32/97) had a perfect 
score. 

Table 1 indicates the total number (and percent) of students who missed each question and then further breaks 
down that number in terms of whether the students were in the calculator or no-calculator group. An 
independent-samples t-test compared the mean total DT scores of the calculator and no-calculator groups, and 
found a statistically significant difference between the two: f(198) = -5.08, p = 0.001. The mean of the 
calculator group was significantly higher (M = 9.40, SI) = 0.84) than the mean of the no-calculator group (M = 
8.49, SD = 1.58); the equivalent of an entire letter grade difference on an A through F scale. Looking 
specifically at the error rates, there were 207 overall errors (out of a possible 2,000). Approximately 30% of the 
207 errors were in the calculator group, and 70% of the errors were in the no-calculator group. These error rates 
included multiple errors made by numerous subjects. 


Table 1. Number of errors and number incorrect per question 


Question 

T ntnl 


Incorrect 

Incorrect 


1 Uldl 

incorrect 

(N=200) 

Percent 

incorrect 

with 

calculator 

(N=103) 

without 

calculator 

(N=97) 

1. Subtract these numbers: 9,864-5,947 

22 

11 

4 

18 

2. Multiply: 92x34 

22 

11 

0 

22 

3. Add the following: $126.30 + $265.12 + 
$196.40 

7 

3.5 

2 

5 

4. An airplane travels 360 kms in three hours. 

15 

7.5 

e. 

9 

How far in one hour? 

D 

5. If a pie is cut into sixths, how many pieces 
would there be? 

15 

7.5 

4 

11 

6. William bought 6 oranges at 5 cents each, and 
had 15 cents left over. How much did he have at 

12 

6 

7 

5 

first? 

7. lane had $2.75. Mary had $ 0.95 more than 
Jane. How much did Jane and Mary have together? 

38 

19 

18 

20 

8. A boy bought a bicycle for $21.50. He sold it 
for $23.75. Did he gain or lose money and by how 
much? 

7 

3.5 

3 

4 

9. Mary’s mother bought a hat for $2.85. What 
was her change from $5? 

15 

7.5 

1 

14 

10. There are 36 children in one classroom and 33 
in another. How much will it cost to buy a crayon. 

54 

27 

16 

38 

at 7 cents each, for each child? 

Total (out of 2,000) 

207 

10.4 

61 

146 


The researchers ran a MANOVA that examined the effect of calculators on each individual question on the DT, 
and found a statistically significant effect: Lambda{\Q,1%9) = 1428.34, p = 0.001. Follow-up univariate 
ANOVAs indicated that calculators proved significantly helpful for Question 1 (/T 1,198) = 11.51, p = 0.001), 
Question 2 (F(l,198) = 29.91, p = 0.001), Question 5 (7^(1,198) = 4.05, p = 0.046), Question 9 CF(1,198) = 
13.82, p = 0.001), and Question 10 1/71,198) = 15.09, p = 0.001). Questions 1, 2, and 9 were straightforward 
arithmetic questions, and the use of calculators all but eliminated these errors. It is difficult to see why Question 
5 discriminated between both groups, as there is no advantage that the calculator could provide. Question 10, 
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the most missed question, involved the use of decimals, and some of the errors involved misplacement of the 
decimal point whereas others involved wrong numbers and blanks. Almost 70% of the errors on this question 
were from the no-calculator group. The use of calculators cut that error rate in half, but there was still a large 
error rate compared to that of other questions, so it is unclear whether computational or conceptual aspects were 
responsible for them. 

The researchers then entered the 12 demographic, predictor variables into a step-wise, multiple linear regression 
to predict students’ total DT score, and found a statistically significant regression equation when using the 
number of high school and college math courses as predictors: ( F( 1,162) = 4.14, p = 0.043). The number of 
math courses barely accounted for 3% of the variance in scores, however (R 2 = 0.03). Four of the demographic 
variables were significantly correlated with DT scores: liking mathematics (r = .23, p < .001), self-assessment 
of high school mathematics ability (r = .24, p < .01), self-assessment of general mathematic ability (r = .17, p < 
.05), and interest in STEM (r = .25, p < .01). These significant correlations, however, together accounted for 
less than 10% of the variability in the total scores, and were so inter-correlated that they may simply be tapping 
into the same construct. 


Discussion 

The purpose of this experiment was to measure the level of mathematical preparation of current college 
students, and to test whether calculators (which Standing (2006) did not allow) improve mathematical 
performance in such students as it does in precollege students. Standing (2006) found that only 33% of college 
students could obtain a perfect score on a third grade arithmetic test from 1932. A higher proportion of our 
sample - 45% - obtained a perfect score on the D’Amore Test, but the higher overall percentage of perfect 
scores was almost entirely a function of better performance from the calculator group. Our study explicitly 
tested whether completing such a test with or without calculators would affect the results, thereby also 
identifying whether the types of errors students make in mathematical problems today are conceptual or 
computational. We identified 56% perfect scores for the calculator group vs. 33% perfect scores for the no¬ 
calculator group (exactly that of Standing’s, who did not allow calculators). Prohibiting the use of calculators 
underestimates the mathematical performance of college students. Using calculators improves mathematical 
performance in college students as it does with precollege students (Ellington, 2003; Hembree & Dessart, 1986). 
We did not find a similar liking of mathematics or STEM areas or self-rated mathematical ability, however. 

One of the performance aspects the researchers explored was the specific type of errors made by the 
participants, whether they were computational (e.g., addition, subtraction) or conceptual (e.g., unable to set up 
the problem, unable to determine a method to find the answer). Questions 1, 2, and 9 are clearly computational 
questions as defined above, and it could be argued that Question 10 requires some conceptual understanding. 
With overall error rates of 6% and 15% for the calculator and no-calculator group, respectively, the use of 
calculators clearly cut the computational errors by more than half. Conceptual errors still remained, but the total 
scores clearly benefited when calculators were permitted. Translated into grades, the difference amounted to the 
calculator group earning an average of an A (94%) vs. the no-calculator group earning an average of a B (86%) 
on the test. The current study did not confirm Standing’s (2006) finding that year in school or self-rated 
mathematical ability predicts performance on the DT, and because our sample consisted mostly of psychology 
majors the researchers did not analyze whether major was a significant predictor. The researchers were able to 
confirm his findings, however, that age, sex, GPA, liking of STEM areas, quality of high school mathematics 
instruction, and ease with computers did not predict performance on the DT. Our results also suggest more than 
just small differences in mathematical performance when calculators are used as contrasted with the results 
found by Scheuneman et al. (2002) and especially Bridgeman et al. (1995) on the SAT. 

There were two main limitations to this study. There was not a diverse sample with regard to major, sex, and 
ethnicity. Moreover, the DT is not constructed to clearly separate conceptual mathematical errors from 
computational ones, and thus this issue needs further, careful exploration. 

This study suggests that computational errors are easily reduced by allowing students to use a calculator. 
Changes in the use of technology today, especially in hand-held devices, may well translate into students today 
not memorizing the multiplication tables or being unaware of the process of multiplying two or more digit 
numbers. Educators’ initial reaction is to assume that modern students are thus not ‘learning’ because they are 
not using the traditional methods. As this study has shown, however, this conclusion needs to be examined very 
carefully. The fact that students do not learn the way that previous generations did, is not evidence for lack of 
learning, or any of the other dire conclusions that are often drawn. The new methods may be just as effective as 
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the traditional ones, and more research needs to be conducted to determine which methods are most useful in 
today’s world. One way to determine that would be to focus on the difference between computational and 
conceptual errors. A different test tapping concepts in mathematics would more accurately distinguish between 
both types of errors, and be able to address this question more explicitly and comprehensively. 

Relying on research rather than traditional guidelines to govern the uses of calculators in the classroom would 
have important implications in teaching. Some might suggest that calculators are simply easier, but we strongly 
suggest that studies be done to determine how, specifically, students are currently learning this material. If 
students are taught with calculators - as currently recommended by the Common Core Standards - and have 
access to calculators on hand-held devices, the effect could be one of exposure, namely, learning with 
calculators as a part of their environment, as opposed to difficulty, and would thus justify the need to use 
calculators so as to match how math is taught with how it is assessed (Loyd, 1991). Teachers of Statistics and 
other courses who use quantitative data could then develop pedagogically sound policies that would focus on 
conceptual mastery of the material, and recognize that current and future students bring very different 
experiences and tools to the classroom than previous generations did. It may be time for faculty to re-think 
calculators in order to maintain sound teaching practices. 
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Appendix 

The D’Amore Test 

You may write on the front and back of this test sheet. 

1. Subtract these numbers: 9,864-5,947 

2. Multiply: 92 x 34 

3. Add the following: $126.30 + $265.12 + 196.40 

4. An airplane travels 360 kilometers in three hours. How far will it go in one hour? 

5. If a pie is cut into sixths, how many pieces would there be? 

6. William bought six oranges at 5 cents each and had 15 cents left over. How much did he have at first? 

7. Jane had $2.75. Mary had 95 cents more than Jane. How much did Jane and Mary have together? 

8. A boy bought a bicycle for $21.50. He sold it for $23.75. Did he gain or lose money and by how 
much? 

9. Mary’s mother bought a hat for $2.85. What was her change from $5? 

10. There are 36 children in one classroom and 33 in another classroom. How much will it cost to buy a 
crayon, at 7 cents each, for each child? 



